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(54) Title: IDENTIFICATION OF OBJECTS BY MEANS OF MULTIPLE IMAGING 

(57) Abstract 

Detection of fetal nucleated red blood cells (NR- 
BCs) is achieved by employing a combination of bright- 
field and fluorescence images of nuclear and cytoplas- 
mic markers. The brightfield and fluorescence images 
are all obtained with a single multi-bandpass dichroic 
mirror. The objects in the sample are stained with a flu- 
orescent dye that selectively stains nuclei and a dye that 
selectively stains fetal hemoglubin in the cytoplasm of 
fetal RBCs. UV excitation provides fluorescent emis- 
sions from the stained cell nuclei and visible illumi- 
nation provides brightfield transmission of light that is 
absorbed by the stained cytoplasm. The images are 
processed to determine regions where the fluorescent 
emissions by cell nuclei in response to the UV exci- 
tation and the absorption by fetal hemoglobin of the 
brightfield illumination overlap or are in close proxim- 
ity. The brightfield and fluorescence images may be 
sequentially acquired or derived from a single image 
where the UV excitation and visible illumination occur 
simultaneously. 
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IDENTIFICATION OF OBJECTS BY MEANS OF MULTIPLE IMAGING 



CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority from U.S. Provisional 
Application No. 60/046,470, filed 05/14/97, of Ilya Ravkin for 
"IDENTIFICATION OF TARGET FEATURES SUCH AS NUCLEATED RED BLOOD CELLS USING 
FLUORESCENCE AND TRANSMISSION SCANNING, " the entire disclosure of which, 
5 including all attached documents and appendices, is incorporated by 
reference for all purposes. 

BACKGROUND OF THE INVENTION 
The present invention relates generally to optical recognition 
:10 systems, and- more particularly to identifying fetal nucleated erythrocytes 
from maw^rnal cells in a blood sample of a pregnant woman. 

It is known to perform image processing on images of 
biological samples where different dyes are caused to reside on different 
portions of the sample, each dye residing on a particular feature that 
15 characterizes an object of interest. The sample is then illuminated and 
imaged in a manner that the different features of interest can be 
distinguished from each other and from the background. This typically 
entails acquiring separate images using the appropriate light sources, 
filters, and optical setup so that each image's particular type of feature 
20 appears recognizably. 

SUMMARY OF THE INVENTION 
The present invention provides robust and efficient techniques 
for analyzing samples to find objects of interest that are interspersed 

25 with other objects. 

In short, a sample is prepared to impart optical properties to 
objects so that objects of interest, when imaged on an imaging medium 
under a plurality of different illumination schemes, exhibit a combination 
of features that is different from combinations of features exhibited by 

30 other objects. This combination is therefore referred to as the unique 
combination. The objects of interest are found by analyzing images 
arising from the respective illumination schemes to determine instances 
where the unique combination of features from the different images meets a 
predetermined proximity constraint (e.g., overlap or near-overlap) . 

35 The illumination schemes and the optical properties are such 

that the sample is imaged on the imaging medium under the plurality of 
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different illumination schemes without having to move any optical elements 
into or out of the path between the sample and the imaging medium. Thus, 
images corresponding to the illumination schemes are largely immune to 
registration problems, thereby making the overlap of the unique 
combination of features a reliable indication of an object of interest. 

The images can be separately acquired images, each taken with 
a respective one of the illumination schemes, or can be derived from a 
lesser number of images, each taken with a combination of the illumination 
schemes in effect simultaneously. 

In a specific example: the objects of interest are fetal 
nucleated red blood cells (NRBCs) and the other objects include non- 
nucleated red blood cells (RBCs) and nucleated white blood cells (WBCs) ,- 
the objects in the sample are stained with a fluorescent dye that 
selectively stains nuclei and a dye that selectively stains fetal 
15 hemoglobin in the cytoplasm of fetal RBCs; there are two different 
illumination schemes, namely UV excitation to provide fluorescent 
emissions from the stained cell nuclei and brightfield transmission of 
light that is absorbed by the stained cytoplasm; and the unique 
combination of features is the fluorescent emissions by cell' 'ht&slbi in 
20"-— • response to the UV excitation and the absorperon l>y "fetal - hemoglobin of 
the brightfield illumination. 

In specific embodiments, the image that is expected to contain 
the fewest features is analyzed first to determine candidate regions of 
interest for subsequent processing. This is accomplished by finding 
25 regions in the image that contain the feature that objects of interest (as 
well as other objects) exhibit in the image, and then examining the 
corresponding regions in the images to determine whether or not the images 
all contain instances of the respective features in sufficient proximity 
to denote the presence of an object of interest. 
30 In specific embodiments, the images (or preferably candidate 

regions thereof) are processed to generate respective contrast masks, 
which are combined and further processed to provide the locations of 
objects of interest in the images. The combination can include a logical 
AND operation between the masks, possibly themselves morphologically 
35 dilated to form a seed image. The further processing can include 

reconstructing the seed within the masks to provide the desired regions 
representing the features of* interest where they denote the presence of an 
object of interest. 

A further understanding of the nature and advantages of the 
40 present invention may be realized by reference to the remaining portions 
of the specification and the drawings. 



45 



BRIEF DESCRIPTION OP THE DRAWINGS 
FIG, 1 is an optical schematic of apparatus according to 
embodiment of the invention; 



Best Available Copy 



WO 98/52018 PCT/US98/09689 

3. 

FIG. 2 is a block diagram of a computer system according to an 
embodiment of the invention; 

FIG, 3 is a flowchart showing the sequence of operations 
performed in specific embodiments of the invention; 
5 FIG. 4A is a flowchart showing image generation steps using 

simultaneous illumination of the sample; 

FIG. 4B is a combined light intensity profile through a fetal 
nucleated red blood cell (NRBC) ; 

FIG. 5 is a flowchart showing steps for determining regions of 

10 interest; 

FIG. 6 is a flowchart showing steps for constructing two 
separate masks that represent nuclear contrast and cytoplasmic contrast; 
and 

FIG. 7 shows a representative set of data that is generated 
15 and stored for each NRBC, 



20 



30 



35 



DESCRIPTION OF SPECIFIC EMBODIMENTS 

1 • Imaging Overview 



The present invention utilizes fluorescence iraaging in 
connection with brightfield (transmission) imaging to locate certain 
objects of interest in a sample. . In the particular application, the 
samples are blood smears from pregnant women (9-16 weeks gestation) 
25 enriched for nucleated red blood cells (NRBCs) , which are the objects of 
interest . 

In the particular implementation, the invention is carried out 
to identify such objects for further analysis such as FISH. FISH stands 
for Fluorescence In-Situ Hybridization. A FISH sample is prepared by 
using a probe that binds to a particular DNA sequence in the chromosomes 
in the sample, and the probe is labeled with a fluorescent dye. M-FISH 
refers to using multiple probes, each of which binds to a different DNA 
sequence in the chromosomes in the sample, where each probe is labeled 
with a different dye or combination of two or more dyes. For that reason, 
the apparatus described below has additional functionality beyond what is 
needed for carrying out the invention. 

A given fluorescent dye is characterized by an excitation 
(absorption) spectrum and an emission spectrum. The excitation and 
emission spectra are also sometimes referred to as the excitation and 
40 emission bands. Accordingly when the dye is irradiated with light at a 

wavelength within the excitation band, the dye fluoresces, emitting light 
at wavelengths in the emission band. Thus when the sample is irradiated 
with excitation radiation in a frequency band that excites a given dye, 
portions of the sample to which the probe labeled with the given dye is 
45 attached fluoresce. If the light emanating from the sample is filtered to 
reject light outside the given dye's emission band, and then imaged, the 
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image nominally shows only those portions of the sample that bind the 
probe labeled with the given dye. 

2. The Sample 

5 

Although the details of the sample preparation are not part of 
the invention, the preparation will be described briefly to provide 
context for the description of the specific embodiments. The present 
invention is directed to the second phase of a three-phase evaluation that 
10 includes enrichment of fetal NRBCs from maternal blood, positive 

identification of fetal NRBCs, and genetic analysis of fetal NRBCs. As 
mentioned above, in specific implementations, the apparatus is configured 
so as to be suitable to play a role in both the second phase 
(identification) and third phases (genetic analysis) . 
15 The enrichment procedure starts with 20mL of EDTA 

anticoagulated whole blood from pregnant women at 9 to 16 weeks post 
conception. It is calculated from PCR studies [Bianchi97] that this 
amount of blood contains approximately from 20 to 100 fetal cells. The 
goal is to achieve a 10,000-fold reduction of maternal blood cells without 
significant- k loss,of the target NBRGs. This will permit the deposition of - 
the fetal NRBC on a slide. The first step of enrichment is the separation 
of the blood sample into density fractions by centrifugation of the whole 
blood in a specially designed plastic tube !Saunders95] . The NRBCs are 
harvested from the interface layer between white blood cells (WBCs) and 
25 red blood cells (RBCs) . The second step is a selective lysis of the 

remaining maternal RBCs. The third and final step separates NRBCs from 
the remaining WBCs in a three-layer density gradient formed of a silica 
colloid Percoll (Pharmacia, Uppsala, Sweden) suspended in gelatin under 
hypertonic conditions. After centrifugation, the NRBCs are harvested from 
30 the bottom of the gradient and deposited on a slide. 

The resulting slide contains NRBCs, RBCs, and WBCs, all of 
them of possible fetal or maternal origin. In embodiments of the 
invention, a set of features that identify the fetal NRBCs are created to 
distinguish them from other types of cells. This is done by creating one 
35 type of contrast in cells containing fetal hemoglobin, and another type of 
contrast in cells having a nucleus. The slide is first reacted with the 
primary antibody, namely mouse anti fetal hemoglobin (HbF) , then with the 
secondary antibody, namely goat anti mouse conjugated to biotin, and 
lastly streptavidin conjugated with alkaline phosphatase is added followed 
40 by Vector Blue substrate. The result is a blue precipitate on the 

cytoplasm of cells that contain fetal hemoglobin. A DNA intercalating 
agent (DAPI) gives all nuclei a fluorescent blue stain; the presence of 
both these contrasts determines a fetal NRBC. 

Custom slides were developed for use with the invention. 
These slides have four painted squares in the corners with crosses laser 
etched in each square. Prior to scanning, coordinates of the reference 
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points are recorded in the scan data file. At any time later, the 
reference points can be easily found and centered in the camera field of 
view. The offset is used for accurate relocation to all other objects in 
the scan file. 

5 

3. Optical System 



FIG. 1 is a schematic showing a representative microscope 
system 5 for carrying out combined fluorescence (preferably epi- 

10 illuminated) and brightfield (transmission) imaging of a sample 10 

according to the present invention. The sample is shown mounted to a 
stage 11, which provides three-dimensional rectilinear motion. As will be 
described below, the invention can be implemented with sequential 
acquisition of brightfield and fluorescent images, or with simultaneous 

15 acquisition of brightfield and fluorescent image components, with 

subsequent separation into separate images. The optical system includes 
portions for each of the two types of images, although a significant 
portion of the optics are shared. 

The portion of -the optical train for epi- illuminated 

2,0 f luorescence imaging . includes an excitation f ilter 12 (shown as-one^of ^ 
several such filters on a filter wheel) , a polychroic mirror 15, a 
microscope objective 17 (say 10-100x) , and an emission filter 20. The 
present invention can be implemented with a single excitation filter, a 
dichroic mirror, and an emission filter with a single passband. However, 

25 it is preferred to perform additional fluorescent measurements (FISH) , and 
therefore it is preferred to use a multiband emission filter and a 
polychroic mirror so that multiple fluorescent images can be acquired 
without changing the mirror or emission filter. 

Excitation radiation from an ultraviolet (UV) source 25 passes 

30 through excitation filter 12, is largely reflected by mirror 15 and 

proceeds through the microscope objective 17 to sample 10. The excitation 
light traveling toward the sample is shown schematically by hollow 
arrowheads. Fluorescent radiation emitted from sample 10 passes back 
through objective 17, through mirror 15, and through emission filter 20 to 

35 form an image in an image plane 30. The fluorescent light traveling away 
from the sample is shown schematically by solid black arrowheads. The 
image is digitized by a CCD video camera 32 and the digitized image is 
sent to a computer 35 for subsequent processing. Computer 35 is also used 
to control various components in the system, as will be described below. 

40 Polychroic mirror 15 and emission filter 20 are typically 

rigidly mounted to a supporting structure 40 (shown in phantom) ; the 
assembly is often referred to as a cube, with multiple cubes being movable 
into and out of the optical path. Oppositely directed arrows 42 represent 
a suitable mechanism such as a rotatable turret or a detented slide 

45 mechanism. The multiple excitation filters are typically deployed on a 
rotatable filter wheel (as shown) . In a typical microscope, objective 
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lens 17 will be one of several mounted on a turret or like structure. 
This is shown schematically by oppositely directed arrows 43. 

The portion of the optical train for the brightfield imaging 
includes a visible light source 45, a passband filter 47 (shown as one of 
several such filters on a filter wheel), a mirror 50, and a condenser 52. 
Illumination radiation from source 45 passes through passband filter 47, 
is reflected by a mirror 50 to condenser 52. The radiation passing 
through the condenser illuminates sample 10, and proceeds through 
microscope objective 17. The illumination radiation is in a wavelength 
range that passes through polychroic mirror 15 and emission filter 20. 
The visible illumination light is shown schematically by hatched 
arrowheads . 

The system also includes a series of motor and lamp 
controllers, shown as a single controller block 55 interfaced to computer 

15 35. Controller 55 controls various of the elements shown in the optical 
schematic of PIG. 1. This includes controlling motors for microscope 
focus, the filter wheels for excitation filter 12 and transmission filter 
47, sample stage 11, cube turret 42, and objective turret 43. Controller 
100 also controls UV light source 25 and visible light source 45. It 
«30~ .^should be -understood^ however, that- the ^ invent ion 5 does-not^reqUire "tfia^ y 
all the controllable elements be under computer control, although it is 
preferred to have the controls mentioned. The particular techniques for 
interfacing a computer to external instruments are known to those skilled 
in the art, and since they are not, per ^e, part of the invention, they 

25 will not be described further. 

In specific implementations, the microscope is an Olympus BX- 
60 microscope (Olympus America, Inc., Melville, NY), and includes 
transmission and fluorescence capabilities, a trinocular head, and lOx, 
20x, and 40x objectives. Stage 11 can be a single or a multi-slide 

30 scanning stage (Maerzhauser Co., Upper Saddle River, NJ) , and is mounted 
on the microscope with a 7 -position transmission filter wheel, a 12- 
poBition fluorescence filter wheel, and a focus drive (TOFRA, Palo Alto, 
CA) . These devices are driven by stepping motors and are controlled by 
microstepping motor controllers (Intelligent Motion Systems, Marlborough, 

35 CT) . Video camera 32 has light integration capability <COHU 4910, Cohu, 
Inc., San Diego, CA) , and is coupled to a frame grabber board, which 
includes a 10-bit ADC and frame averaging. UV source 25 is typically a 
mercury arc lamp while visible light source 47 is typically a halogen 
lamp . 

40 As will be described in greater detail below, the optical 

configuration of PIG. 1 is used for the sequential or simultaneous 
detection of DAPI fluorescence and Vector Blue absorption, and 
subsequently also for FISH imaging. An important advantage is the 
permanent presence of the polychroic mirror and emission filter for the 

45 detection of both absorption and fluorescence. 
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In the specific implementation, the epi- illumination starts 
with the mercury arc, traverses the DAPI excitation filter and is 
reflected down by the polychroic mirror; excites the blue fluorescence of 
the DAPI stained cells, the emitted light returns through the objective, 
and passes through both the mirror and the emission filter to the camera. 
The trans -illumination starts with the halogen lamp, passes through a 
long-pass (red) filter, is absorbed by the cells stained with Vector Blue, 
and passes through the objective, the polychroic mirror and emission 
filter to the camera. Depending on the DNA probes, a Chroma 83000 triple 
band filter set (Chroma Technology Corp., Brattleboro, VT) or a Vysis quad 
DAPI/Aqua/Green/Orange filter set (Vysis, Inc., Downers Grove, IL) is 
used. Transmission Kohler illumination and the spectral filter optimize 
contrast for the cytoplasmic color on the slide 



20 



15 4. Computer System 

FIG. 2 is a simplified block diagram of computer 35. The 
computer is used to control all motion, perform image acquisition and 
processing, and user interface functions. In a specific implementation, 
computer 35 ? % a a workstation running the Windows 95 operating systei;: 

In accor^ce^with^toq 
the computer system includes a process.;!- 60 tbpfr communicates with a 
number of peripheral devices via a bus subsystem 65. These peripheral 
devices typically include a memory subsystem 67, an input facility 70, a 
display subsystem 72, output devices such as a printer 73, and a file 
25 storage system 75. 

In this context, the term "bus subsystem" is used generically 
so as to include any mechanism for letting the various components of the 
system communicate with each other as intended. With the exception of 
some of the input devices and the display, the other components need not 

30 be at the same physical location. Thus, for example, portions of the file 
storage system could be connected via various local-area or wide-area 
network media, including telephone lines. Similarly, the input devices 
and display need not be at the same location as the processor, although it 
is anticipated that the present invention will most often be implemented 

35 in the context of PCs and workstations. 

Bus subsystem 65 is shown schematically as a single bus, but a 
typical system has a number of buses such as a local bus and one or more 
expansion buses (e.g., ADB, SCSI, ISA, EISA, MCA, NuBus, or PCI), as well 
as serial and parallel ports. Network connections are usually established 

40 through a device such as a network adapter on one of these expansion buses 
or a modem on a serial port . The computer system may be a desktop system 
or a portable system. 

Memory subsystem 67 includes a number of memories including a 
main random access memory (RAM) 80 and a read only memory (ROM) 82 in 

45 which fixed instructions are stored. In the case of Macintosh -compatible 
personal computers the ROM would include portions of the operating system; 
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in the case of IBM- compatible personal computers, the ROM would include 
the BIOS (basic input/output system) . 

Input facility 70 typically includes a user input device such 
as a keyboard 90, and may further include a pointing device 92 and a 
5 scanner 93. The pointing device may be an indirect pointing device such 
as a mouse, trackball, touchpad, or graphics tablet, or a direct pointing 
device such as a touchscreen incorporated into the display. Other types 
of user input devices, such as voice recognition systems, are also 
possible. Camera 32 can be considered part of the input facility. 

10 Display subsystem 72 typically includes a display controller 

94 and a display device 95 coupled to the display controller. The display 
device may be a cathode ray tube (CRT) , a flat -panel device such as a 
liquid crystal display (LCD), or a projection device. Display controller 
provides control signals to the display device and normally includes a 

15 display memory (not shown in the figure) for storing the pixels that 

appear on the display device. The display subsystem may also provide non- 
visual display such as audio output. 

The file Btorage system provides persistent (non-volatile) 
storav? for program and data files, and typically includes at l*ast one 
-.wt *-, 2 JW*, ,^^^dk^driye t 9,6 . .and . at. JLeas.t one floppy disk-..driye^9/7 > . (with., associated 
removable media) . There may alE^ be utfcar devices such as a CD-ROM drive 
98 and optical drives (all with their associate removable media) . 
Additionally, the system may include drives of the type with removable 
media cartridges. The removable media cartridges may, for example be hard 

25 disk cartridges, such as those marketed by Syquest, Iomega, and others, 

and flexible disk cartridges, such as those marketed by Iomega and others. 
As noted above, one or more of the drives may be located at a remote 
location, such as in a server on a local area network or at a site on the 
Internet's World Wide Web. 



30 



5. Process ing Overview 



FXG« 3 is a flowchart showing, at a high level, the sequence 
of operations performed in specific embodiments of the invention. As 
35 mentioned above, the invention is directed toward finding specific objects 
of interest in an sample. In the specific application, the objects of 
interest are fetal nucleated red blood cells (NRBCs) in a sample of 
maternal blood. Despite enrichment, the number of NRBCs is likely to be 
small compared to the number of non-nucleated red blood cells (RBCs) and 
40 nucleated white blood cells (WBCs) . 

The analysis begins at a step 120 of generating images with 
two different contrasts, one which shows regions of cytoplasm and one 
which shows nuclei. As mentioned above, these two contrasts arise from 
the absorption of transmitted visible (red) light by the (blue) dye that 
preferentially stains cytoplasm and the fluorescence from cell nuclei 
stained by the fluorescent dye DAPI. Further, as mentioned above, the two 
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images can be generated from separately acquired images resulting from 
sequential exposure of the sample to the two light sources, or can be 
generated from a single acquired image with the sample illuminated by both 
light sources simultaneously. 
5 The images are generated for each of a plurality of fields on 

the slide. A field corresponds to the number of pixels on the CCD imaging 
array, typically 512x512, and there are typically hundreds or thousands of 
fields on the slide. This is accomplished by stepping sample stage 11 
along both axes in the plane of the sample. Since the objects of interest 

10 are likely to occupy a relatively small fraction of the image field, it is 
preferred to perform a step 125 of quickly determining the portions of 
each field that are candidates for containing objects of interest. The 
subsequent processing steps in the method are recited as applying to the 
images, but it should be understood that the processing steps are 

15 typically only applied to the candidate regions of each field. 

The individual images are processed separately at a step 130 
to provide derivative images (masks) that can be correlated or combined at 
a step 135. to determine if an object of interest is present. The method 
has thus proved the locations of regions of the sample that are expected 
^-^^f^SfiBS^^J^!^ 9 .^ interest, making^it possible, to ^subject^the objects— 
to further analysis at a step 140. For example, In a subsequent analysis 
phase, the computer can control the stage to step the sample to these 
locations for manual inspection or further computerized processing. Thus, 
the result of the method is acceptance of an image as falling into a 

25 specific class of object, which is to be counted or to be examined in more 
detail for other contrasts, for example other colors of fluorescence for 
FISH analysis. 



6. Image Acquisition 

6-1 Sequential Image Acquisition 

With sequential illumination of the sample, separate 
brightfield and fluorescence images are acquired on every field. For the 
absorption image, epi- illumination is blocked by an opaque segment of the 
excitation filter wheel, and visible light source 45 sends red light 
through the specimen to the camera. The absorption image shows objects 
positive for antibody staining. For the fluorescence image, 
transillumination is blocked, and the specimen is excited by light from UV 
source 25 going down through the objective. The fluorescence image shows 
all nuclei in the field. Fetal NRBCs are the objects having dark 
cytoplasms in the absorption image and bright nuclei in the fluorescence 
image . 



45 



6-2 Simultaneous Image Acquisition 

FIG. 4A is a flowchart showing steps that can be performed to 
implement step 120 of FIG. 3 using simultaneous illumination of the 
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sample- The reference number 120' is used to denote this variant. Also 
shown is a highly stylized view of a small portion of the images at each 
step. These stylized views show three levels, black, white, and gray (as 
represented by cross-hatching) . The objects are shown as having a 
5 circular shape. 

At a step 145, a single image is acquired with the sample 
being trans illuminated with red light and epi- illuminated with UV light at 
the same time. The intensities of both illumination systems light are 
balanced so that the empty background corresponds to a mid- level gray. 

10 Antibody- stained objects (cytoplasm) appear darker than the background and 
DAPI- stained objects (nuclei) appear brighter than the background. FIG. 
4B is a combined light intensity profile through a fetal NRBC, and shows a 
significant drop in the cytoplasm, a rise in the nucleus, and another drop 
in the cytoplasm. In the schematic representation, the image shows an 

15 NRBC having cytoplasm (black in the figure) surrounding nucleus (white in 
the figure) , and another nucleus without surrounding cytoplasm (white in 
the figure) , all against a gray background (hatched in the figure) . 

Since the two images have been combined optically, they need 
to be separated digitally. As a first step to thia ]^id, a background gray 
**«v *>* *sr . .«*?#20'~* «- level- is determined 'at a step 150v*'^Iir* vsp^eific'^mpiementatlbn^^'the^mb'de*; 

is used. Thus, the whole fieiu is measured and a histogram of the number 
of pixels at each possible intensity level is constructed. The histogram 
is smoothed by adjacent averaging and the intensity corresponding to the 
top of the highest peak in the histogranuis defined as the background 

25 value of light intensity. It would also be possible to use an average 
pixel value to specify the mid- level gray. 

The combined images are separated at a step 155 into a below- 
the -background component, characterizing antibody absorption, and an 
above- the -background component, characterizing DAPI fluorescence. This is 

30 accomplished by comparing the background value determined in step 150 to 
the image on a pixel -by -pixel basis. This process is similar to a 
subtraction with saturation. This produces separate positive and 
negative-going images, which are essentially the two separate contrasts 
dissected from the single image. 

35 This process can be viewed, with reference to FIG. 4B, as 

follows. For the absorption image, add a value corresponding to the 
difference between the maximum intensity value (255) and the background 
intensity value to all pixels. This has the effect of bringing the 
cytoplasm trough regions up from near-black to mid-gray, bringing the 

40 background up to a white or near-white level, and clipping the 

fluorescence peak at white. For the fluorescence image, subtract the 
background intensity value from all pixels, thereby bringing the 
fluorescence peak down to mid-gray, bringing the background down to a 
black or near-black level, and clipping the cytoplasm regions to black. 

45 
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6.3 Relative Merits 

Simultaneous illumination and image acquisition has the 
advantages of speed (because there is only one image to acquire and no 
filter switch for every field) , and easy human recognition of NRBCs under 
5 the microscope. On the other hand, the disadvantage is a tighter 

requirement for cell fixation and sharpness of the DAPI image. There are 
conditions when the nuclear counterstain image is fuzzy and overlaps with 
the cytoplasm. In this case, simultaneous illumination cancels out both 
contrasts and the sequential method should be used. It should be 

10 realized, however, that the optical configuration is still advantageous 
for the sequential image acquisition since none of the elements in the 
optical path from the sample to the camera have to be moved, thus 
enhancing the ease of registration of the separate images for further 
processing. Naturally, where the two images were generated from a single 

15 combined image, they are automatically registered. 



7 . Image Processing and Correlation 



7.1 Find Regalia Potentially Co ntaining Objects of Interest 

^nJ^%^S^^S^°^j.Jt^J^1S^^B09^ to limit .further^ref ined, 
J> - ocessing to those regions of the image (or ima^'i f ield) r / that are 
considered likely to contain objects of interest. The NRBCs in the sample 
have a nucleus surrounded by cytoplasm containing fetal hemoglobin. In 
the images, this translates to an overlap or close proximity of an 

25 absorption feature and a fluorescence feature. In order to find 

corresponding regions in the respective images containing both features, 
it is necessary to scan one of the images for that image's distinctive 
feature. Once the scan of one image yields a set of candidate regions, it 
is only necessary to scan corresponding regions in the other image to 

30 determine whether they contain the other imaged distinctive feature. 

This will be more efficient if the image that is expected to contain fewer 
features is scanned first. For example, if there are significantly more 
absorption features in the absorption image than fluorescent features in 
the fluorescence image, the fluorescence image should be scanned for 

35 features first. This order can be determined by the user, or set 

automatically as a function of the type of scan to be performed. The 
relative frequency of object types typically depends on the sample 
preparation (e.g., degree of enrichment). 

FIG. 5 is a flowchart showing steps that can be performed to 

40 implement step 125 of FIG* 3. At this point in the overall process, 

separate absorption and fluorescence images have been provided, either 
having been acquired from separate optical images or having been 
electronically separated from a single optical image as shown in FIGS. 4A 
and 4B. The image that is expected to have fewer of itB characteristic 

45 features will be referred to as the "first" image. 
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At a step 160, a background is determined for the first image 
based on all pixels in the first image. If the first image is the 
fluorescent image, the background value is set equal to the value of the 
darkest (lowest intensity value) pixel in the image. At a step 162, this 
background is subtracted from all the pixels in the image. At a step 165, 
all pixels are compared to a threshold for the image, to generate a binary 
image. A pixel is considered to be "ON" if it is above the threshold 
(fluorescent image) or below the threshold (absorption image) ; The 
threshold is relatively high, since at this stage it is only desired to 
see whether features are present without attempting to determine their 
precise outlines. 

At a step 167, the image is analyzed to determine the 
locations of sufficiently large connected components (i.e., instances of a 
sufficient number of connected ON pixels) to signify the likely presence 
of an object. This is done using blob analysis according to techniques 
known to those skilled in the art. The regions surrounding the connected 
components in the first image define a set of candidate regions. These 
regions are significantly smaller than the 512x512 field, typically the 
smallest convenient square of pixels that can be defisSti, say 32x32 pixels 
, A _pr 64^4^pA».l8^.^At a step. 170, , .coordinates^fpr; theaaejfc. or ...candidate 
regions are stored for f urther analysis . As mentioned above, further 
processing is preferably restricted to these regions. 

7.2 Mask Creation 

FIG. 6 is a flowchart showing steps that can be performed to 
implement steps 130 and 135 of PIG. 3, namely separately processing the 
two images for a set of features which leads to the construction of two 
separate masks that represent nuclear contrast and cytoplasmic contrast. 
Input at this stage are the cytoplasm and nuclear images generated in step 
120, with backgrounds having been subtracted at steps 162 and 172. As 
noted above, the further processing is done only on the image regions 
identified in step 125. 

At a step 180, the images are subjected to further background 
subtraction operations, but this time the background is determined on a 
region-by- region basis. At a step 182, the images are thresholded to 
create binary masks. For the cytoplasm image, pixels darker than the 
threshold are set to ON (white in the figure) and pixels lighter than the 
threshold are set to OFF (black in the figure) . For the nuclear image, 
pixels lighter than the threshold are set to ON (white) and pixels darker 
than the threshold are set to OFF (black) . 

When viewing the grayscale image (s), the notion of whether the 
features in the image (s) overlap tends to be rather subjective. Once the 
images have been thresholded to generate binary images, the separation is 
directly quantifiable. Depending on the threshold used to create the 
binary images and the objects' geometries, features that appear 
subjectively to an observer to overlap in the grayscale images may be 
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disjoint, although in close proximity, in the binary images. It is then 
possible to define a maximum separation of the features (in the binary 
images) that will be still be considered to represent an overlap, and thus 
a candidate for the object of interest. 

The specific feature of the cells of interest is the overlap 
or close proximity of antibody-positive and DAPI-positive areas. In order 
to find these overlaps or proximities, the binary masks are subjected at a 
step 185 to a first set of morphological image processing steps [Serra89] . 
In particular, the images are morphologically "opened" to eliminate small 
random signals which are not part of the identified localized feature. 
Opening is in effect a filtering process. The opened masks are then 
morphologically "dilated, " which has the effect of expanding each 
remaining feature (region of ON pixels, shown as white in the figure) by a 
designated amount in all directions. Note that the dilation has the 
effect of shrinking holes (regions of OFF pixels surrounded by or 
intruding into regions of ON pixels. 

The designated amount of expansion is determined by the 
morphological dilation operation. More specifically, the degree to which 
an edge of a rejlon of ON pixels is expanded by dilation is determined by 
_ the~size>ypf .what, is^ref erred .rto -as r»a^ structuring element . - -Thus- the size 
of the structuring element is selected in view of t hi /amount by which 
features are allowed to be separated and still be considered to "overlap." 
This is a function of image magnification. It has been found that a 
structuring element that moves the boundaries by one pixel operates well 
at magnifications of 10x-20x. This means that two objects that just touch 
will, when dilated, have an overlap of two pixels. 

An appropriate size for the structuring element can be 
empirically determined in a straightforward manner. For example, a 
representative sample of objects where the features appear to overlap in 
the grayscale images can be subjected to the thresholding, and the 
distribution of separations in the binary images determined. It has 
further been found that the results are not very sensitive to the size of 
the structuring element. For example, an expansion by two pixels would 
also likely be suitable. 

At a step 190, the opened, dilated masks are combined or 
overlapped on the same field to discover true overlapping features. This 
is a logical AND operation that leaves only the rim of where the two masks 
overlap. This is also referred to as the intersection of the two masks. 
At a step 195, the intersection mask is morphologically dilated to produce 
a seed image. At a step 200, the seed image is used to finally identify 
the true cytoplasm and nuclear regions in the two images. This is done by 
reconstructing the seed within the cytoplasm and nuclear contrast images 
that were input to step 185. As can be seen in the figure, only the 
nucleus that had the surrounding cytoplasm appears in the final mask. 
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8. Workflow and Performance 

The workflow for analysis of slides consists of three major 
steps: scan, review, and probe acquisition (FISH). The first step starts 
5 with a slide and produces a data file, which contains images and 

measurements of detected objects. In the second step, the operator 
reviews results of the scan, classifies and annotates the cells, and 
selects cells of interest for probe acquisition. The review is done 
primarily from the screen images, but the slide is available for immediate 
10 relocation and inspection under the microscope. Finally, probe images of 
the selected cells are acquired and the probe spots are counted. 

The main form of presenting information for review is in a 
data grid, which contains images, measured features, classification, and 
annotation. FIG. 7 shows a representative set of data that is generated 
15 and stored for each NRBC. Ratio of Axes is the ratio of the short main 
inertia axis to the long main inertia axis, normalized to the range [0- 
100] . Small values are typical for elongated shapes; large values are 
typical for objects that appear more nearly circular in the image. 
Compactess is the ratio of the moment of iifertia of a disk with the same 

hi ^ ^iafcw u 2 O.yi^-v trArfi.a-j.as ><fche > given g,hape * to - the- moment v- p f > ~-ine r t i a- of the** shaped " Ai This r at io s -~ • <* * -^-a*****^ 

is normalized uo the r~x»ge [0-100] . Small values are typical for ring-' 
like, star-like or irregular shapes, large values are typical for compact 
and round objects. 

To set up a scan the user performs the following actions: 

25 

Defines the scan area to correspond to the actual smear 
location on the slide. 

Selects a representative field, and adjusts the image 
digitization parameters for each of the filters used in scanning. 
30 (Sometimes the digitizer adjustment is done on calibration slides) . 

Runs the finding algorithm on the selected field and adjusts 
the parameters if needed. 

Sets the position of the reference point to provide for 
35 accurate relocation in the future. 

Gives a name to the scan, and starts the scan. 

The scan can be interrupted at any time to look at the objects and 
readjust parameters before resuming the scan. After the scan is complete, 

40 the user returns for review. 

The objects can be revisited any number of times and each time 
a group of several images can be acquired for each object (for example, 
original images from scanning, same filter images at higher magnification, 
and probe images) . Each group of images may be shown differently (e.g., 

45 in monochrome, in color, or both) . The review process is aided by two- 
dimensional distribution plots of any two of the measured features. Each 
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class in the distribution is shown in a different color and the display 
resembles that in flow cytometry. Switching of views among stored cell 
images, distribution of cells in the feature space, and microscope 
relocation is instantaneous. 

Another useful facility is sorting by linear combination of 
the measured features, which presents cells for review in the order of 
most likely candidates first. The result of the review is a 
classification and selection of cells for probe acquisition.- At this 
time, the finding algorithms are designed to look for objects of one class 
(defined by the algorithm and its parameters) , and there is no further 
automatic classification in the feature space. Classification is done 
manually by the user. 

Searching for one combination of features, namely the overlap 
of DAPI fluorescence and HbF antibody absorption, has proved to be robust, 
15 missing few cells (<3%) . The false positives are the overlaps of RBCs, 
WBCs, and debris. To separate NRBCs from false positives, the following 
parameters are used: average and integrated intensity in nucleus and 
cytoplasm, and area and shape of the nucleus and of the union of nucleus 
*ud cytoplasm. 

— > • 

^.^>..^^ 20 ...^ ^3^£? *?°S>S ca ™ xi2l 9-.«U.4 ^«MXPA&*l t smejar^is t i/<i -.hour. J at 

magnification lOx) . The av-rage relocation error is 2 mm (4 mm maximum) . 
The program finds 97.2% of the NRBCs found by a trained operator. 
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40 

10. Conclusion 



In conclusion, it can be seen that the present invention 
provides methods and apparatus for efficiently and reliably identifying 
objects of interest in an image. A system according to the invention 
provides, in particular implementations, automated finding of cells, 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

SI BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

U BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

U LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



WO 98/52018 



PCT/US98/09689 



16. 

storage of cell images and their slide coordinates, with interactive 
review, classification, and selection by the operator. The invention is 
robust in the face of the high variation among specimens, and adapts to 
difference in staining intensity and probe brightness, type of cell 
deposition on slides (smear or cytospin) , cell density, different kinds of 
samples, and prevalence of different cell types. 

While the above is a complete description of specific 
embodiments of the invention, various modifications, alternative 
constructions, and equivalents may be used. For example, while it is 
preferred to have epi- illumination for the UV excitation, it is possible 
to use transillumination for the UV excitation. Similarly, while the 
specific embodiment is based around a standard microscope, it is possible 
to carry out the invention with a specially configured optical setup. 

Therefore, the above description should not be taken as 
limiting the scope of the invention as defined by the claims. 
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WHAT IS CLAIMED Tfl ■ 

1 1. A method of finding objects of interest that are 

2 interspersed with other objects in a sample, the method comprising: 

3 providing a sample where objects have optical properties so 

4 that objects of interest, when imaged on an imaging medium under a 

5 plurality of different illumination schemes, exhibit a combination of 

6 features, referred to as the unique combination, that is different from 

7 combinations of features exhibited by other objects; 

8 subjecting the sample to the plurality of illumination 

9 schemes ; 

10 generating a corresponding plurality of images; and 

11 analyzing the plurality of images to determine instances where 

12 the unique combination of features from the different images meets a 

13 predetermined proximity constraint; 

14 the sample being imaged on the imaging medium under the 

15 plurality of different illumination schemes without moving any optical 

16 elements into or out of the path between the sample and the imaging 

17 medium. 

1 2- The method of .;Iaim 1 wherein the plurality of images are 

2 acquired separately with only a respective one of the plurality of 

3 illumination schemes being in effect at a given time. 

1 3. The method of claim 1 wherein the plurality of images are 

2 derived from a lesser number of images, each of the lesser number of 

3 images being taken with a combination of the illumination schemes in 

4 effect simultaneously. 

1 4. The method of claim 3 wherein first and second images are 

2 derived from single image by: 

3 configuring the illumination schemes corresponding to the 

4 first and second images so that features corresponding to the first and 

5 second illumination scheme are respectively at intensities above and below 

6 a background level; and 

7 generating the first image by reducing each pixel value in the 

8 single image by a first amount so that background regions are generally at 

9 minimum intensity; and 

10 generating the second image by increasing each pixel value in 

11 the single image by a second amount so that background regions are 

12 generally at maximum intensity. 

1 5. The method of claim 1 wherein analyzing the plurality of 

2 images comprises: 

3 examining a particular image of the plurality of images to 

4 find a set of regions of the particular image that contains features; and 
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5 analyzing only the corresponding regions of the plurality of 

6 images . 

1 6. The method of claim 1 wherein analyzing the plurality of 

2 images comprises: 

3 applying a threshold to each image to create a respective 

4 binary mask having ON pixels in areas representing the presence of 

5 features in that image; 

6 morphologically processing each mask to expand regions of 

7 connected ON pixels; and 

8 logically ANDing the masks to define regions of overlap or 

9 near overlap of features present in the images. 

1 7. The method of claim 1 wherein: 

2 the objects of interest are fetal nucleated red blood cells 

3 (NRBCs) and the other objects include non-nucleated red blood cells (RBCs) 

4 and nucleated white blood cells (WBCs) ; 

5 the objects in the sample are stained with a fluorescent dye 

6 that selectively stains mclei and a dye that selectively stains fetal 

• - 7 hemoglobin in the cytoplasm of fetal RBCs; ; .< ^. 

8 the plurality of different illumination schemes*- include UV 

9 excitation to provide fluorescent emissions from the stained cell nuclei 

10 and brightfield transmission of light that is absorbed by the stained 

11 cytoplasm; and 

12 the unique combination of features is the fluorescent 

13 emissions by cell nuclei in response to the UV excitation and the 

14 absorption by fetal hemoglobin of the brightfield illumination. 

1 8. A method of detecting fetal nucleated red blood cells in a 

2 sample region that contains additional objects, the method comprising: 

3 providing a sample stained with a first dye that 

4 preferentially stains fetal hemoglobin in the cytoplasm of fetal RBCs and 

5 a second dye that preferentially stains cell nuclei; 

6 transilluminating the sample with first light in a wavelength 

7 range that is absorbed by the first dye; 

8 simultaneously illuminating the sample with second light that 

9 causes the second dye to fluoresce and emit third light; 

10 forming a single image of the first light passing through the 

11 sample and the third light emitted by the sample; and 

12 processing the single image to identify cells that have both 

13 stained cytoplasm and nuclei. 

1 9. The method of claim 8 wherein processing the single image 

2 includes the substeps of: 

3 determining a background level for the single image; 
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4 generating a nuclear image by reducing pixel values in the 

5 single image by a first amount so that background regions are generally at 

6 minimum intensity; 

7 generating a cytoplasm image by increasing pixel values in the 

8 single image by a second amount so that background regions are generally 

9 at maximum intensity; 

10 processing the cytoplasm image to produce a cytoplasm contrast 

11 mask having regions representing the presence of cytoplasm; 

12 processing the nuclear image to produce a nuclear contrast 

13 mask having regions representing the presence of cell nuclei; 

14 combining the cytoplasm contrast mask and the nuclear contrast 

15 mask to determine regions of connected ON pixels in the cytoplasm contrast 

16 mask that are sufficiently close to regions of connected ON pixels in the 

17 nuclear contrast mask to signify the likely presence of fetal nucleated 

18 red blood cells. 

1 10- The method of claim 9 wherein processing the nuclear and 

2 cytoplasm images comprises: 

3 examining one of the nuclear and cycoplasm images to find a 

-.y. .i:.„. '} .*:fi9- t rf ^..-3^^SP a 0 f the one of the nuclear and^cytoplasm images^that . 

5 contains featu^ec; aiiu 

6 processing only the corresponding regions of the other of the 

7 nuclear and cytoplasm images . 

1 11. The method of claim 9 wherein the cytoplasm contrast mask 

2 the nuclear contrast mask are combined by: 

3 morphologically processing each mask to expand regions of 

4 connected ON pixels; and 

5 logically ANDing the masks, as morphologically processed, to 

6 define regions of overlap or near overlap of features present in the 

7 images . 

1 12. The method of claim 8 wherein processing the single image 

2 includes : 

3 producing a cytoplasm contrast mask having regions 

4 representing the presence of cytoplasm and a nuclear contrast mask having 

5 regions representing the presence of cell nuclei; 

6 morphologically processing each mask to expand regions of 

7 connected ON pixels; 

8 performing a logical AND between the cytoplasm contrast mask 

9 and the nuclear contrast mask to produce an intersection image; and 

10 processing the intersection image and the cytoplasm contrast 

11 mask and the nuclear contrast mask to provide a mask that specifies areas 

12 containing cytoplasm stained with the first dye and a cell nucleus stained 

13 with the second dye. 
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1 13. A method of detecting fetal nucleated red blood cells in 

2 a sample region that contains additional objects, the method comprising: 

3 providing a sample stained with a first dye that 

4 preferentially stains cytoplasm and a second dye that preferentially 

5 stains cell nuclei; 

6 transilluminating the sample with first light in a wavelength 

7 range that is absorbed by the first dye; 

8 forming a first image of the first light passing through the 

9 sample ; 

10 processing the first image to produce a cytoplasm contrast 

11 mask having regions representing the presence of cytoplasm; 

12 illuminating the sample with second light that causes the 

13 second dye to fluoresce and emit third light; 

14 forming a second image of the third light emitted by the 

15 sample; 

16 processing the second image to produce a nuclear contrast mask 

17 having regions representing the presence of cell nuclei; 

18 combining the cytoplasm contrast mask and the nuclear contrast 

19 mask to determine regions c£ connected ON pixels in the cytoplasm contrast 
^aftsmr^. *.^,*> f .ii:2Q: •.. mask that are sufficiently close to regions of ; connected ON pixels in the 

2i " nuciat ; contrast mask to signify the likely presence of fetal nucleated 

22 red blood cells. 

1 14. The method of claim 13 wherein the first and second 

2 images are acquired separately with only a respective one of the first 

3 light and second light illuminating the sample at a given time. 

1 15. The method of claim 13 wherein the first and second 

2 images are derived from a single image taken with the first light and 

3 second light illuminating the sample simultaneously. 

1 16. The method of claim 15 wherein the first and second 

2 images are derived from the single image by: 

3 configuring the illumination of the first and second light so 

4 that features corresponding to fluorescence and absorption are 

5 respectively at intensities above and below a background level; and 

6 generating the first image by reducing pixel values in the 

7 single image by a first amount so that background regions are generally at 

8 minimum intensity; and 

9 generating the second image by increasing pixel values in the 

10 single image by a second amount so that background regions are generally 

11 at maximum intensity. 



1 

2 



17. The method of claim 13 wherein processing the first image 
to produce a cytoplasm contrast mask includes applying a threshold to the 
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3 first image to create a binary mask having ON pixels in areas representing 

4 the presence of features in that image, 

1 18. The method of claim 13 wherein combining the cytoplasm 

2 contrast mask and the nuclear contrast mask comprises: 

3 morphologically processing each mask to expand regions of 

4 connected ON pixels, thereby producing processed masks; and 

5 logically ANDing the processed masks to produce an 

6 intersection image that defines regions of overlap or near overlap of 

7 features present in the images. 

1 19, The method of claim 18, and further comprising: 

2 morphologically processing the intersection image to expand 

3 regions of connected ON pixels, thereby producing a seed image; and 

4 reconstructing the seed image within the cytoplasm contrast 

5 mask and the nuclear contrast mask to represent cytoplasm and nuclei for 

6 those objects that are likely to be fetal nucleated red blood cells . 

1 20. A method of detecting feual nucleated red blood cells in 

u — .^<w...\ ^2.^^ .a.., sample region that contains additional objects, the^method comprising : • r- -. 

3 providing £ sample stained with a first dye that ' 

4 preferentially stains fetal cytoplasm and a second dye that preferentially 

5 stains cell nuclei; 

6 transilluminating the sample: with first light in a wavelength 

7 range that is absorbed by the first dye; 

8 simultaneously illuminating the sample with second light that 

9 causes the second dye to fluoresce and emit third light; 

10 forming a forming a single image of the first light passing 

11 through the sample and the third light emitted by the sample; 

12 determining a background level for the single image; 

13 generating a cytoplasm image by increasing pixel values in the 

14 single image by a second amount so that background regions are generally 

15 at maximum intensity; 

16 thresholding the cytoplasm image to create a binary cytoplasm 

17 contrast mask having regions having ON pixels representing the presence of 

18 objects stained by the first dye; 

19 generating a nuclear image by reducing pixel values in the 

20 single image by a first amount so that background regions are generally at 

21 minimum intensity; 

22 thresholding the nuclear image to create a binary nuclear 

23 contrast mask having regions having ON pixels representing the presence of 

24 objects stained by the second dye; 

25 morphologically processing each mask to expand regions of 

26 connected ON pixels, thereby producing processed masks; 

27 logically ANDing the processed masks to produce an 

28 intersection image that defines regions of overlap or near overlap of 
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29 objects in the cytoplasm image stained by the first dye and objects in the 

30 nuclear image stained by the second dye; 

31 morphologically processing the intersection image to expand 

32 regions of connected ON pixels, thereby producing a seed image; and 

33 reconstructing the seed image within the cytoplasm contrast 

34 mask and the nuclear contrast mask to represent cytoplasm and nuclei for 

35 those objects that are likely to be fetal nucleated red blood cells. 
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